Systems and methods for augmented reality

ABSTRACT

Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. An augmented reality display system comprises a handheld component housing an electromagnetic field emitter, the electromagnetic field emitter emitting a known magnetic field, the head mounted component coupled to one or more electromagnetic sensors that detect the magnetic field emitted by the electromagnetic field emitter housed in the handheld component, wherein a head pose is known, and a controller communicatively coupled to the handheld component and the head mounted component, the controller receiving magnetic field data from the handheld component, and receiving sensor data from the head mounted component, wherein the controller determining a hand pose based at least in part on the received magnetic field data and the received sensor data.

CROSS-REFERENCE TO RELATED U.S. PATENT APPLICATIONS

The present disclosure is a continuation of pending U.S. patentapplication Ser. No. 17/137,107, filed Dec. 29, 2020 and entitled“SYSTEMS AND METHODS FOR AUGMENTED REALITY,” under attorney docketnumber ML-0247USCON1, which is a continuation of U.S. patent applicationSer. No. 15/062,104, filed Mar. 5, 2016 and entitled “SYSTEMS ANDMETHODS FOR AUGMENTED REALITY,” under attorney docket numberML.20031.00, which claims priority to U.S. Provisional PatentApplication Ser. No. 62/128,993, filed Mar. 5, 2015 and entitled“ELECTROMAGNETIC TRACKING SYSTEM AND METHOD FOR AUGMENTED REALITY,”under attorney docket number ML.30031.00, and also claims priority toU.S. Provisional Patent Application Ser. No. 62/292,185, filed Feb. 5,2016 and entitled “SYSTEMS AND METHODS FOR AUGMENTED REALITY,” underattorney docket number ML.30062.00. The entire contents of theaforementioned patent applications are hereby explicitly incorporated byreference into the present disclosure for all purposes.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality” or “augmentedreality” experiences, wherein digitally reproduced images or portionsthereof are presented to a user in a manner wherein they seem to be, ormay be perceived as, real. A virtual reality, or “VR”, scenariotypically involves presentation of digital or virtual image informationwithout transparency to other actual real-world visual input. Anaugmented reality, or “AR”, scenario typically involves presentation ofdigital or virtual image information as an augmentation to visualizationof the actual world around the user.

For example, referring to FIG. 1, an augmented reality scene 4 isdepicted wherein a user of an AR technology sees a real-world park-likesetting 6 featuring people, trees, buildings in the background, and aconcrete platform 1120. In addition to these items, the user of the ARtechnology may also perceive a robot statue 1110 standing upon thereal-world platform 1120, and a cartoon-like avatar character 2 flyingaround the park. Of course, the virtual elements 2 and 1110 do not existin the real world, but the user perceives these virtual objects as beingpart of, and as interacting with objects of the real world (e.g., 6,1120, etc.). It should be appreciated, the human visual perceptionsystem is very complex, and producing such an AR scene that facilitatesa comfortable, natural-feeling, rich presentation of virtual imageelements amongst other virtual or real-world imagery elements ischallenging.

For instance, head-worn AR displays (e.g., helmet-mounted displays, orsmart glasses) may be coupled to a user's head, and thus may move whenthe user's head moves. If the user's head motions are detected by thedisplay system, the data being displayed can be updated to take thechange in head pose into account. The head pose may be utilized toappropriately render virtual content to the user. Thus any smallvariation may disrupt and/or diminish the delivery or timing of virtualcontent that is delivered to the user's AR display.

As an example, if a user wearing a head-worn display views a virtualrepresentation of a three-dimensional (3-D) object on the display andwalks around the area where the 3-D object appears, that 3-D object canbe re-rendered for each viewpoint, giving the user the perception thathe or she is walking around an object that occupies real space. If thehead-worn display is used to present multiple objects within a virtualspace (for instance, a rich virtual world), measurements of head pose(i.e., the location and orientation of the user's head) can be used tore-render the scene to match the user's dynamically changing headlocation and orientation, and provide an increased sense of immersion inthe virtual space. Similarly, when a user of AR technology isinteracting with the virtual world, he or she may use an object orhis/her hand to point to objects or to select options. In order for thisinteraction to occur, localization of the object or the user's hand toan accurate degree is also important. Thus, both head pose, and “handpose” are both crucial, and localization techniques must be used inorder to accurately depict virtual content to the user.

In AR systems, detection and/or calculation of head pose and/or handpose can facilitate the AR display system to render virtual objects suchthat they appear to occupy a space in the real world in a manner that iscongruent to the objects of the real world. Presenting an AR scenerealistically such that the virtual content does not seemjarring/disorienting in relation to one or more real objects improvesthe user's enjoyment of the AR experience. In addition, detection of theposition and/or orientation of a real object, such as a handheld device(which also may be referred to as a “totem”), haptic device, or otherreal physical object, in relation to the user's head or AR system mayalso facilitate the display system in presenting display information tothe user to enable the user to interact with certain aspects of the ARsystem efficiently.

It should be appreciated that in AR applications, placement of virtualobjects in spatial relation to physical objects (e.g., presented toappear spatially proximate a physical object in two or three dimensions)is a non-trivial problem. For example, head movement may significantlycomplicate placement of virtual objects in a view of an ambientenvironment. This may be true whether the view is captured as an imageof the ambient environment and then projected or displayed to the enduser, or whether the end user perceives the view of the ambientenvironment directly. For instance, head movement may cause the field ofview of the user to change. This may, in turn, require an update towhere various virtual objects are displayed in the field of view of theend user. Similarly, movement of the hand (in case of a handheld object)when used to interact with the system provides the same challenge. Thesemovements may be fast and typically need to be accurately detected andlocalized at a high refresh rate and low latency.

Additionally, head and/or hand movements may occur at a large variety ofranges and speeds. The speed may vary not only between different typesof head movements, but within or across the range of a single movement.For instance, speed of head movement may initially increase (e.g.,linearly or otherwise) from a starting point, and may decrease as anending point is reached, obtaining a maximum speed somewhere between thestarting and ending points of the head movement. Rapid movements mayeven exceed the ability of the particular display or projectiontechnology to render images that appear uniform and/or as smooth motionto the end user.

Head or hand tracking accuracy and latency (i.e., the elapsed timebetween when the user moves his or her head/hand and the time when theimage gets updated and displayed to the user) have been challenges forVR and AR systems. Especially for display systems that fill asubstantial portion of the user's visual field with virtual elements, itis critical that the accuracy of tracking is high and that the overallsystem latency is very low from the first detection of motion to theupdating of the light that is delivered by the display to the user'svisual system. If the latency is high, the system can create a mismatchbetween the user's vestibular and visual sensory systems, and generate auser perception scenario that can lead to motion sickness or simulatorsickness. If the system latency is high, the apparent location ofvirtual objects may appear unstable during rapid head motions.

In addition to head-worn display systems, other display systems can alsobenefit from accurate and low-latency head pose detection. These mayinclude head-tracked display systems in which the display is not worn onthe user's body, but is, e.g., mounted on a wall or other surface. Thehead-tracked display may act like a window onto a scene, and as a usermoves his head relative to the “window” the scene is re-rendered tomatch the user's changing viewpoint. Other systems may include ahead-worn projection system, in which a head-worn display projects lightonto the real world.

Additionally, in order to provide a realistic AR experience, AR systemsmay be designed to be interactive with the user. For example, multipleusers may play a ball game with a virtual ball and/or other virtualobjects. One user may “catch” the virtual ball, and throw the ball backto another user. In another embodiment, a first user may be providedwith a totem (e.g., a physical “bat” communicatively coupled to the ARsystem) to hit the virtual ball. In other embodiments, a virtual userinterface may be presented to the AR user to allow the user to selectone of many options. The user may use totems, haptic devices, wearablecomponents, or simply touch the virtual screen to interact with thesystem.

Detecting a pose and an orientation of the user (e.g., the user's headand hand), and detecting a physical location of real objects in spacemay enable the AR system to display virtual content in an effective andenjoyable manner. However, such accurate detection of head and hand posemay be difficult to achieve. In other words, the AR system mustrecognize a physical location of a real object (e.g., user's head,totem, haptic device, wearable component, user's hand, etc.) andcorrelate the physical coordinates of the real object to virtualcoordinates corresponding to one or more virtual objects being displayedto the user. This process can be improved by highly accurate sensors andsensor recognition systems that track a position and orientation of oneor more objects at rapid rates. Current approaches do not performlocalization at satisfactory speed or precision standards.

There, thus, is a need for a better localization system in the contextof AR and VR devices.

SUMMARY

Embodiments of the present invention are directed to devices, systemsand methods for facilitating virtual reality and/or augmented realityinteraction for one or more users.

In one aspect, an augmented reality (AR) display system comprises anelectromagnetic field emitter to emit a known magnetic field, anelectromagnetic sensor to measure a parameter related to a magnetic fluxmeasured at the electromagnetic sensor as a result of the emitted knownmagnetic field, wherein world coordinates of the electromagnetic sensorare known, a controller to determine pose information relative to theelectromagnetic field emitter based at least in part on the measureparameter related to the magnetic flux measured at the electromagneticsensor, and a display system to display virtual content to a user basedat least in part on the determined pose information relative to theelectromagnetic field emitter.

In one or more embodiments, the electromagnetic field emitter resides ina mobile component of the AR display system. In one or more embodiments,the mobile component is a hand-held component. In one or moreembodiments, the mobile component is a totem.

In one or more embodiments, the mobile component is a head-mountedcomponent of the AR display system. In one or more embodiments, the ARdisplay system further comprises a head-mounted component that housesthe display system, wherein the electromagnetic sensor is operativelycoupled to the head-mounted component. In one or more embodiments, theworld coordinates of the electromagnetic sensor is known based at leastin part on SLAM analysis performed to determine head pose information,wherein the electromagnetic sensor is operatively coupled to ahead-mounted component that houses the display system.

In one or more embodiments, the AR display further comprises one or morecameras operatively coupled to the head-mounted component, and whereinthe SLAM analysis is performed based at least on data captured by theone or more cameras. In one or more embodiments, the electromagneticsensors comprise one or more inertial measurement units (IMUs).

In one or more embodiments, the pose information corresponds to at leasta position and orientation of the electromagnetic field emitter relativeto the world. In one or more embodiments, the pose information isanalyzed to determine world coordinates corresponding to theelectromagnetic field emitter. In one or more embodiments, thecontroller detects an interaction with one or more virtual contentsbased at least in part on the pose information corresponding to theelectromagnetic field emitter.

In one or more embodiments, the display system displays virtual contentto the user based at least in part on the detected interaction. In oneor more embodiments, the electromagnetic sensor comprises at least threecoils to measure magnetic flux in three directions. In one or moreembodiments, the at least three coils are housed together atsubstantially the same location, the electromagnetic sensor beingcoupled to a head-mounted component of the AR display system.

In one or more embodiments, the at least three coils are housed atdifferent locations of the head-mounted component of the AR displaysystem.

The AR display system of claim 1, further comprising a control and quickrelease module to decouple the magnetic field emitted by theelectromagnetic field emitter. In one or more embodiments, the ARdisplay system further comprises additional localization resources todetermine the world coordinates of the electromagnetic field emitter. Inone or more embodiments, the additional localization resources comprisesa GPS receiver. In one or more embodiments, the additional localizationresources comprises a beacon.

In one or more embodiments, the electromagnetic sensor comprises anon-solid ferrite cube. In one or more embodiments, the electromagneticsensor comprises a stack of ferrite disks. In one or more embodiments,the electromagnetic sensor comprises a plurality of ferrite rods eachhaving a polymer coating. In one or more embodiments, theelectromagnetic sensor comprises a time division multiplexing switch.

In another aspect, a method to display augmented reality comprisesemitting, through an electromagnetic field emitter, a known magneticfield, measuring, through an electromagnetic sensor, a parameter relatedto a magnetic flux measured at the electromagnetic sensor as a result ofthe emitted known magnetic field, wherein world coordinates of theelectromagnetic sensor are known, determining pose information relativeto the electromagnetic field emitter based at least in part on themeasured parameter related to the magnetic flux measured at theelectromagnetic sensor, and displaying virtual content to a user basedat least in part on the determined pose information relative to theelectromagnetic field emitter.

In one or more embodiments, the electromagnetic field emitter resides ina mobile component of the AR display system. In one or more embodiments,the mobile component is a hand-held component. In one or moreembodiments, the mobile component is a totem. In one or moreembodiments, the mobile component is a head-mounted component of the ARdisplay system.

In one or more embodiments, the method further comprises housing thedisplay system in a head-mounted component, wherein the electromagneticsensor is operatively coupled to the head-mounted component. In one ormore embodiments, the world coordinates of the electromagnetic sensor isknown based at least in part on SLAM analysis performed to determinehead pose information, wherein the electromagnetic sensor is operativelycoupled to a head-mounted component that houses the display system.

In one or more embodiments, further comprises capturing image datathrough one or more cameras that are operatively coupled to thehead-mounted component, and wherein the SLAM analysis is performed basedat least on data captured by the one or more cameras. In one or moreembodiments, the electromagnetic sensors comprise one or more inertialmeasurement units (IMUs).

In one or more embodiments, the pose information corresponds to at leasta position and orientation of the electromagnetic field emitter relativeto the world. In one or more embodiments, the pose information isanalyzed to determine world coordinates corresponding to theelectromagnetic field emitter. In one or more embodiments, the methodfurther comprises detecting an interaction with one or more virtualcontents based at least in part on the pose information corresponding tothe electromagnetic field emitter.

In one or more embodiments, the method further comprises displayingvirtual content to the user based at least in part on the detectedinteraction. In one or more embodiments, the electromagnetic sensorcomprises at least three coils to measure magnetic flux in threedirections. In one or more embodiments, the at least three coils arehoused together at substantially the same location, the electromagneticsensor being coupled to a head-mounted component of the AR displaysystem. In one or more embodiments, the at least three coils are housedat different locations of the head-mounted component of the AR displaysystem.

In one or more embodiments, the method further comprises decoupling themagnetic field emitted by the electromagnetic field emitter through acontrol and quick release module. In one or more embodiments, the methodfurther comprises determining the world coordinates of theelectromagnetic field emitter through additional localization resources.In one or more embodiments, the additional localization resourcescomprises a GPS receiver. In one or more embodiments, the additionallocalization resources comprises a beacon.

In yet another aspect, an augmented reality display system, comprises ahandheld component housing an electromagnetic field emitter, theelectromagnetic field emitter emitting a known magnetic field, a headmounted component having a display system that displays virtual contentto a user, the head mounted component coupled to one or moreelectromagnetic sensors that detect the magnetic field emitted by theelectromagnetic field emitter housed in the handheld component, whereina head pose is known, and a controller communicatively coupled to thehandheld component and the head mounted component, the controllerreceiving magnetic field data from the handheld component, and receivingsensor data from the head mounted component, wherein the controllerdetermines a hand pose based at least in part on the received magneticfield data and the received sensor data, wherein the display systemmodifies the virtual content displayed to the user based at least inpart on the determined hand pose.

In one or more embodiments, the handheld component is mobile. In one ormore embodiments, the handheld component is a totem. In one or moreembodiments, the handheld component is a gaming component. In one ormore embodiments, the head pose is known based at least in part on SLAManalysis.

In one or more embodiments, the AR display system further comprises oneor more cameras operatively coupled to the head-mounted component, andwherein the SLAM analysis is performed based at least on data capturedby the one or more cameras. In one or more embodiments, theelectromagnetic sensor comprises one or more inertial measurement units(IMUs).

In one or more embodiments, the head pose corresponds to at least aposition and orientation of the electromagnetic sensor relative to theworld. In one or more embodiments, the hand pose is analyzed todetermine world coordinates corresponding to the handheld component. Inone or more embodiments, the controller detects an interaction with oneor more virtual contents based at least in part on the determined handpose.

In one or more embodiments, the display system displays the virtualcontent to the user based at least in part on the detected interaction.In one or more embodiments, the electromagnetic sensor comprises atleast three coils to measure magnetic flux in three directions. In oneor more embodiments, the at least three coils are housed together atsubstantially the same location. In one or more embodiments, the atleast three coils are housed at different locations of the head-mountedcomponent.

In one or more embodiments, the AR display system further comprises acontrol and quick release module to decouple the magnetic field emittedby the electromagnetic field emitter. In one or more embodiments, the ARdisplay system further comprises additional localization resources todetermine the hand pose. In one or more embodiments, the additionallocalization resources comprises a GPS receiver. In one or moreembodiments, the additional localization resources comprises a beacon.

Additional and other objects, features, and advantages of the inventionare described in the detail description, figures and claims.

Additional and other objects, features, and advantages of the inventionare described in the detail description, figures and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings illustrate the design and utility of various embodiments ofthe present invention. It should be noted that the figures are not drawnto scale and that elements of similar structures or functions arerepresented by like reference numerals throughout the figures. In orderto better appreciate how to obtain the above-recited and otheradvantages and objects of various embodiments of the invention, a moredetailed description of the present inventions briefly described abovewill be rendered by reference to specific embodiments thereof, which areillustrated in the accompanying drawings. Understanding that thesedrawings depict only typical embodiments of the invention and are nottherefore to be considered limiting of its scope, the invention will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIG. 1 illustrates a plan view of an AR scene displayed to a user of anAR system according to one embodiment.

FIGS. 2A-2D illustrate various embodiments of wearable AR devices

FIG. 3 illustrates an example embodiment of a user of a wearable ARdevice interacting with one or more cloud servers of the AR system.

FIG. 4 illustrates an example embodiment of an electromagnetic trackingsystem.

FIG. 5 illustrates an example method of determining a position andorientation of sensors, according to one example embodiment.

FIG. 6 illustrates an example diagram of utilizing an electromagnetictracking system to determine head pose.

FIG. 7 illustrates an example method of delivering virtual content to auser based on detected head pose.

FIG. 8 illustrates a schematic view of various components of an ARsystem according to one embodiment having an electromagnetic transmitterand electromagnetic sensors.

FIGS. 9A-9F illustrate various embodiments of the control and quickrelease module.

FIG. 10 illustrates one simplified embodiment of the AR device.

FIGS. 11A and 11B illustrate various embodiments of placement of theelectromagnetic sensors on the head-mounted AR system.

FIGS. 12A-12E illustrate various embodiments of a ferrite cube to becoupled to the electromagnetic sensors.

FIG. 13A-13C illustrate various embodiments of circuitry of theelectromagnetic sensors.

FIG. 14 illustrates an example method of using an electromagnetictracking system to detect head and hand pose.

FIG. 15 illustrates another example method of using an electromagnetictracking system to detect head and hand pose.

DETAILED DESCRIPTION

Referring to FIGS. 2A-2D, some general componentry options areillustrated. In the portions of the detailed description which followthe discussion of FIGS. 2A-2D, various systems, subsystems, andcomponents are presented for addressing the objectives of providing ahigh-quality, comfortably-perceived display system for human VR and/orAR.

As shown in FIG. 2A, an AR system user 60 is depicted wearing a headmounted component 58 featuring a frame 64 structure coupled to a displaysystem 62 positioned in front of the eyes of the user. A speaker 66 iscoupled to the frame 64 in the depicted configuration and positionedadjacent the ear canal of the user (in one embodiment, another speaker,not shown, is positioned adjacent the other ear canal of the user toprovide for stereo/shapeable sound control). The display 62 may beoperatively coupled 68, such as by a wired lead or wirelessconnectivity, to a local processing and data module 70 which may bemounted in a variety of configurations, such as fixedly attached to theframe 64, fixedly attached to a helmet or hat 80 as shown in theembodiment of FIG. 2B, embedded in headphones, removably attached to thetorso 82 of the user 60 in a backpack-style configuration as shown inthe embodiment of FIG. 2C, or removably attached to the hip 84 of theuser 60 in a belt-coupling style configuration as shown in theembodiment of FIG. 2D.

The local processing and data module 70 may comprise a power-efficientprocessor or controller, as well as digital memory, such as flashmemory, both of which may be utilized to assist in the processing,caching, and storage of data, which may be (a) captured from sensorswhich may be operatively coupled to the frame 64, such as image capturedevices (such as cameras), microphones, inertial measurement units,accelerometers, compasses, GPS units, radio devices, and/or gyros;and/or (b) acquired and/or processed using the remote processing module72 and/or remote data repository 74, possibly for passage to the display62 after such processing or retrieval. The local processing and datamodule 70 may be operatively coupled (76, 78), such as via a wired orwireless communication links, to the remote processing module 72 andremote data repository 74 such that these remote modules (72, 74) areoperatively coupled to each other and available as resources to thelocal processing and data module 70.

In one embodiment, the remote processing module 72 may comprise one ormore relatively powerful processors or controllers configured to analyzeand process data and/or image information. In one embodiment, the remotedata repository 74 may comprise a relatively large-scale digital datastorage facility, which may be available through the internet or othernetworking configuration in a “cloud” resource configuration. In oneembodiment, all data may be stored and all computation may be performedin the local processing and data module, allowing fully autonomous usefrom any remote modules.

Referring now to FIG. 3, a schematic illustrates coordination betweenthe cloud computing assets 46 and local processing assets, which may,for example reside in head mounted componentry 58 coupled to the user'shead 120 and a local processing and data module 70, coupled to theuser's belt 308; therefore the component 70 may also be termed a “beltpack” 70, as shown in FIG. 3. In one embodiment, the cloud 46 assets,such as one or more cloud server systems 110 are operatively coupled115, such as via wired or wireless networking (wireless being preferredfor mobility, wired being preferred for certain high-bandwidth orhigh-data-volume transfers that may be desired), directly to (40, 42)one or both of the local computing assets, such as processor and memoryconfigurations, coupled to the user's head 120 and belt 308 as describedabove. These computing assets local to the user may be operativelycoupled to each other as well, via wired and/or wireless connectivityconfigurations 44, such as the wired coupling 68 discussed below inreference to FIG. 8. In one embodiment, to maintain a low-inertia andsmall-size subsystem mounted to the user's head 120, primary transferbetween the user and the cloud 46 may be via the link between thesubsystem mounted at the belt 308 and the cloud, with the head mountedsubsystem 120 primarily data-tethered to the belt-based subsystem 308using wireless connectivity, such as ultra-wideband (“UWB”)connectivity, as is currently employed, for example, in personalcomputing peripheral connectivity applications.

With efficient local and remote processing coordination, and anappropriate display device for a user, such as the user interface oruser display system 62 shown in FIG. 2A, or variations thereof, aspectsof one world pertinent to a user's current actual or virtual locationmay be transferred or “passed” to the user and updated in an efficientfashion. In other words, a map of the world may be continually updatedat a storage location which may partially reside on the user's AR systemand partially reside in the cloud resources. The map (also referred toas a “passable world model”) may be a large database comprising rasterimagery, 3-D and 2-D points, parametric information and otherinformation about the real world. As more and more AR users continuallycapture information about their real environment (e.g., through cameras,sensors, IMUs, etc.), the map becomes more and more accurate andcomplete.

With a configuration as described above, wherein there is one “model” ofthe world that can reside on cloud computing resources and bedistributed from the cloud server, such a world can be “passable” to oneor more users in a relatively low bandwidth form. This may be preferableto transferring real-time video data or similar complex information fromone AR system to another. The augmented experience of the personstanding near the statue (i.e., as shown in FIG. 1) may be informed bythe cloud-based world model, a subset of which may be passed down to theperson's local display device to complete the view. A person sitting ata remote display device (e.g., a personal computer sitting on a desk),can efficiently download that same section of information from the cloudand have it rendered on the personal computer display. In yet anotherembodiment, yet another user may be present in real-time at the park,and may take a walk in that park, with a friend (e.g., the person at thepersonal computer) joining the user through a shared AR and/or VRexperience. In order to render the park scene to the friend, the ARsystem may detect a location of the street, a location of the trees inthe park, a location of the statue, etc. This location may be uploadedto the passable world model in the cloud, and the friend (at thepersonal computer) can download the portion of the passable world fromthe cloud, and then start “walking along” with the AR user in the park.Of course, in some embodiments, the friend may be rendered as an avatarin the passable world model to the AR user in the park such that the ARuser can walk alongside the virtual friend in the park.

More particularly, in order to capture details of the world such that itcan be passed on to the cloud (and subsequently to other AR users) 3-Dpoints pertaining to various objects may be captured from theenvironment, and the pose (i.e., vector and/or origin positioninformation relative to the world) of the cameras that capture thoseimages or points may be determined. These 3-D points may be “tagged”, orassociated, with this pose information. It should be appreciated thatthere may be a large number of AR systems capturing the same points inany given environment. For example, points captured by a second camera(of a second AR system) may be utilized to determine the head pose ofthe second camera. In other words, one can orient and/or localize asecond camera based upon comparisons with tagged images from a firstcamera. Then, this information may be utilized to extract textures, makemaps, and create one or more virtual copies of the real world.

In one or more embodiments, the AR system can be utilized to captureboth 3-D points and the 2-D images that produced the points. Asdiscussed above, these points and images may be sent out to the cloudstorage and processing resource (e.g., the servers 110 of FIG. 3), insome embodiments. In other embodiments, this information may be cachedlocally with embedded pose information (i.e., the tagged images) suchthat tagged 2-D images are sent to the cloud along with 3-D points. If auser is observing a dynamic scene, the user may also send additionalinformation up to the cloud servers. In one or more embodiments, objectrecognizers may run (either on the cloud resource or on the localsystem) in order to recognize one or more objects in the capturedpoints. More information on object recognizers and the passable worldmodel may be found in U.S. patent application Ser. No. 14/205,126,entitled “SYSTEM AND METHOD FOR AUTMENTED AND VIRTUAL REALITY”, which isincorporated by reference in its entirety herein, along with thefollowing additional disclosures, which related to augmented and virtualreality systems such as those developed by Magic Leap, Inc. of FortLauderdale, Florida: U.S. patent application Ser. No. 14/641,376; U.S.patent application Ser. No. 14/555,585; U.S. patent application Ser. No.14/212,961; U.S. patent application Ser. No. 14/690,401; U.S. patentapplication Ser. No. 13/663,466; and U.S. patent application Ser. No.13/684,489.

In order to capture points that can be used to create the “passableworld model,” it is helpful to accurately know the user's location, poseand orientation with respect to the world. More particularly, the user'sposition must be localized to a granular degree, because it may beimportant to know the user's head pose, as well as hand pose (if theuser is clutching a handheld component, gesturing, etc.). In one or moreembodiments, GPS and other localization information may be utilized asinputs to such processing. Highly accurate localization of the user'shead, totems, hand gestures, haptic devices etc. are desirable inprocessing images and points derived from a particular AR system, andalso in order to displaying appropriate virtual content to the user.

One approach to achieve high precision localization may involve the useof an electromagnetic field coupled with electromagnetic sensors thatare strategically placed on the user's AR head set, belt pack, and/orother ancillary devices (e.g., totems, haptic devices, gaminginstruments, etc.). Electromagnetic tracking systems typically compriseat least an electromagnetic field emitter and at least oneelectromagnetic field sensor. The electromagnetic sensors may measureelectromagnetic fields with a known distribution. Based on thesemeasurements a position and orientation of a field sensor relative tothe emitter is determined.

Referring now to FIG. 4, an example system of an electromagnetictracking system (e.g., such as those developed by organizations such asthe Biosense (RTM) division of Johnson & Johnson Corporation, Polhemus(RTM), Inc. of Colchester, Vermont, and manufactured by Sixense (RTM)Entertainment, Inc. of Los Gatos, California, and other trackingcompanies) is illustrated. In one or more embodiments, theelectromagnetic tracking system comprises an electromagnetic fieldemitter 402 which is configured to emit a known magnetic field. As shownin FIG. 4, the electromagnetic field emitter 402 may be coupled to apower supply 410 (e.g., electric current, batteries, etc.) to providepower to the electromagnetic field emitter 402.

In one or more embodiments, the electromagnetic field emitter 402comprises several coils (e.g., at least three coils positionedperpendicular to each other to produce a field in the x, y and zdirections) that generate magnetic fields. These magnetic fields areused to establish a coordinate space. This may allow the system to map aposition of the sensors 404 in relation to the known magnetic field,which, in turn, helps determine a position and/or orientation of thesensors 404. In one or more embodiments, the electromagnetic sensors 404a, 404 b, etc. may be attached to one or more real objects. Theelectromagnetic sensors 404 may comprise smaller coils in which currentmay be induced through the emitted electromagnetic field. Generally, the“sensor” components 404 may comprise small coils or loops, such as a setof three differently-oriented (i.e., such as orthogonally orientedrelative to each other) coils coupled together within a small structuresuch as a cube or other container, that are positioned/oriented tocapture incoming magnetic flux from the magnetic field emitted by theelectromagnetic emitter 402. By comparing currents induced through thesecoils, and by knowing the relative position and orientation of the coilsrelative to each other, a relative position and orientation of a sensor404 relative to the electromagnetic emitter 402 may be calculated.

One or more parameters pertaining to a behavior of the coils in theelectromagnetic tracking sensors 404 and the inertial measurement unit(“IMU”) components operatively coupled to the electromagnetic trackingsensors 404 may be measured in order to detect a position and/ororientation of the sensor 404 (and the object to which it is attachedto) relative to a coordinate system to which the electromagnetic fieldemitter 402 is coupled. Of course this coordinate system may betranslated into a world coordinate system, in order to determine alocation or pose of the electromagnetic field emitter in the real world.In one or more embodiments, multiple sensors 404 may be used in relationto the electromagnetic emitter 402 to detect a position and orientationof each of the sensors 404 within the coordinate space associated withthe electromagnetic field emitter 402.

It should be appreciated that in some embodiments, head pose may alreadybe known based on sensors on the headmounted component of the AR system,and SLAM analysis performed based on sensor data and image data capturedthrough the headmounted AR system. However, it may be important to knowa position of the user's hand (e.g., a handheld component like a totem,etc.) relative to the known head pose. In other words, it may beimportant to know a hand pose relative to the head pose. Once therelationship between the head (assuming the sensors are placed on theheadmounted component) and hand is known, a location of the handrelative to the world (e.g., world coordinates) can be easilycalculated.

In one or more embodiments, the electromagnetic tracking system mayprovide 3-D positions (i.e., X, Y and Z directions) of the sensors 404,and may further provide location information of the sensors 404 in twoor three orientation angles. In one or more embodiments, measurements ofthe IMUs may be compared to the measurements of the coil to determine aposition and orientation of the sensors 404. In one or more embodiments,both electromagnetic (EM) data and IMU data, along with various othersources of data, such as cameras, depth sensors, and other sensors, maybe combined to determine the position and orientation of theelectromagnetic sensors 404.

In one or more embodiments, this information may be transmitted (e.g.,wireless communication, Bluetooth, etc.) to a controller 406. In one ormore embodiments, pose information (e.g., position and orientation)corresponding to the sensors 404 may be reported at a relatively highrefresh rate to the controller 406. Conventionally, an electromagneticemitter 402 may be coupled to a relatively stable and large object, suchas a table, operating table, wall, or ceiling, etc. and one or moresensors 404 may be coupled to smaller objects, such as medical devices,handheld gaming components, totems, frame of the head-mounted AR system,or the like.

Alternatively, as described below in reference to FIG. 6, variousfeatures of the electromagnetic tracking system may be employed toproduce a configuration wherein changes or deltas in position and/ororientation between two objects that move in space relative to a morestable global coordinate system may be tracked. In other words, aconfiguration is shown in FIG. 6 wherein a variation of anelectromagnetic tracking system may be utilized to track position andorientation changes between a head-mounted component and a hand-heldcomponent, while head pose relative to the global coordinate system (sayof the room environment local to the user) is determined otherwise, suchas by simultaneous localization and mapping (“SLAM”) techniques usingoutward-capturing cameras which may be coupled to the head mountedcomponent of the AR system.

Referring back to FIG. 4, the controller 406 may control theelectromagnetic field emitter 402, and may also capture measurement datafrom the various electromagnetic sensors 404. It should be appreciatedthat the various components of the system may be coupled to each otherthrough any electro-mechanical or wireless/Bluetooth means. Thecontroller 406 may also store data regarding the known magnetic field,and the coordinate space in relation to the magnetic field. Thisinformation may then be used to detect the position and orientation ofthe sensors 404 in relation to the coordinate space corresponding to theknown electromagnetic field, which can then be used to determined worldcoordinates of the user's hand (e.g., location of the electromagneticemitter).

One advantage of electromagnetic tracking systems is that they canproduce highly accurate tracking results with minimal latency and highresolution. Additionally, the electromagnetic tracking system does notnecessarily rely on optical trackers, thereby making it easier to tracksensors/objects that are not in the user's line-of-vision.

It should be appreciated that the strength of the electromagnetic field(“v”) drops as a cubic function of distance (“r”) from a coiltransmitter (e.g., electromagnetic field emitter 402). One or morealgorithms may be formulated based on a distance of the sensors from theelectromagnetic field emitter. The controller 406 may be configured withsuch algorithms to determine a position and orientation of thesensor/object at varying distances away from the electromagnetic fieldemitter. Given the rapid decline of the strength of the electromagneticfield as one moves farther away from the electromagnetic emitter,improved results, in terms of accuracy, efficiency and low latency, maybe achieved at closer distances. In typical electromagnetic trackingsystems, the electromagnetic field emitter is powered by electriccurrent (e.g., plug-in power supply) and has sensors located within a 20ft. radius away from the electromagnetic field emitter. A shorter radiusbetween the sensors and field emitter may be more desirable in manyapplications, including AR applications.

Referring now to FIG. 5, an example flowchart describing a functioningof a typical electromagnetic tracking system is briefly described. At502, a known electromagnetic field is emitted. In one or moreembodiments, the electromagnetic field emitter may generate a magneticfield. In other words, each coil of the emitter may generate an electricfield in one direction (e.g., x, y or z). The magnetic fields may begenerated with an arbitrary waveform. In one or more embodiments, eachof the axes may oscillate at a slightly different frequency.

At 504, a coordinate space corresponding to the electromagnetic fieldmay be determined. For example, the controller 406 of FIG. 4 mayautomatically determine a coordinate space around the electromagneticemitter based on parameters of the electromagnetic field. At 506, abehavior of the coils at the sensors (which may be attached to a knownobject) may be detected/measured. For example, a current induced at thecoils may be measured. In other embodiments, a rotation of a coil, orother quantifiable behavior may be tracked and measured. At 508, thismeasurement may be used to determine/calculate a position andorientation of the sensor(s) and/or known object. For example, thecontroller may consult a mapping table that correlates a behavior of thecoils at the sensors to various positions or orientations. Based onthese calculations, the position and orientation of the sensors (orobject attached thereto) within the coordinate space may be determined.In some embodiments, the pose/location information may be determined atthe sensors. In other embodiment, the sensors communicate data detectedat the sensors to the controller, and the controller may consult themapping table to determined pose information relative to the knownmagnetic field (e.g., coordinates relative to the handheld component).

In the context of AR systems, one or more components of theelectromagnetic tracking system may need to be modified in order tofacilitate accurate tracking of mobile components. As described above,tracking the user's head pose and orientation is helpful in many ARapplications. Accurate determination of the user's head pose andorientation allows the AR system to display the right virtual content tothe user in the appropriate position in the AR display. For example, thevirtual scene may comprise a monster hiding behind a real building.Depending on the pose and orientation of the user's head in relation tothe building, the view of the virtual monster may need to be modifiedsuch that a realistic AR experience is provided.

In other embodiments, a position and/or orientation of a totem, hapticdevice or some other means of interacting with a virtual content may beimportant in enabling the AR user to interact with the AR system. Forexample, in many gaming applications, the AR system must detect aposition and orientation of a real object in relation to virtualcontent. Or, when displaying a virtual interface, a position of a totem,user's hand, haptic device or any other real object configured forinteraction with the AR system must be known in relation to thedisplayed virtual interface in order for the system to understand acommand, etc. Conventional localization methods including opticaltracking and other methods are typically plagued with high latency andlow resolution problems, which makes rendering virtual contentchallenging in many AR applications.

In one or more embodiments, the electromagnetic tracking system,discussed above may be adapted to the AR system to detect position andorientation of one or more objects in relation to an emittedelectromagnetic field. Typical electromagnetic systems tend to havelarge and bulky electromagnetic emitters (e.g., 402 in FIG. 4), whichmay make them less-than-ideal for use in AR applications. However,smaller electromagnetic emitters (e.g., in the millimeter range) may beused to emit a known electromagnetic field in the context of the ARsystem.

Referring now to FIG. 6, an electromagnetic tracking system may beincorporated into an AR system as shown, with an electromagnetic fieldemitter 602 incorporated as part of a hand-held controller 606. In oneor more embodiments, the hand-held controller may be a totem to be usedin a gaming application. In other embodiments, the hand-held controllermay be a haptic device that may be used to interact with the AR system(e.g., via a virtual user interface). In yet other embodiments, theelectromagnetic field emitter may simply be incorporated as part of thebelt pack 70, as shown in FIG. 2D. The hand-held controller 606 maycomprise a battery 610 or other power supply that powers theelectromagnetic field emitter 602.

It should be appreciated that the electromagnetic field emitter 602 mayalso comprise or be coupled to an IMU component 650 that is configuredto assist in determining position and/or orientation of theelectromagnetic field emitter 602 relative to other components. This maybe useful in cases where both the electromagnetic field emitter 602 andthe sensors 604 (discussed in further detail below) are mobile. In someembodiments, placing the electromagnetic field emitter 602 in thehand-held controller rather than the belt pack, as shown in theembodiment of FIG. 6, ensures that the electromagnetic field emitterdoes not compete for resources at the belt pack, but rather uses its ownbattery source at the hand-held controller 606.

In one or more embodiments, electromagnetic sensors 604 may be placed onone or more locations on the user's headset 58, along with other sensingdevices such as one or more IMUs or additional magnetic flux capturingcoils 608. For example, as shown in FIG. 6, sensors 604, 608 may beplaced on either side of the head set 58. Since these sensors 604, 608are engineered to be rather small (and hence may be less sensitive, insome cases), it may be important to include multiple sensors in order toimprove efficiency and precision of the measurements.

In one or more embodiments, one or more sensors 604, 608 may also beplaced on the belt pack 620 or any other part of the user's body. Thesensors 604, 608 may communicate wirelessly or through Bluetooth® with acomputing apparatus 607 (e.g., the controller) that determines a poseand orientation of the sensors 604, 608 (and the AR headset 58 to whichthey are attached) in relation to the known magnetic field emitted bythe electromagnetic field emitter 602. In one or more embodiments, asshown in FIG. 6, the computing apparatus 607 may reside at the belt pack620. In other embodiments, the computing apparatus 607 may reside at theheadset 58 itself, or even the hand-held controller 604. In one or moreembodiments, the computing apparatus 607 may receive the measurements ofthe sensors 604, 608, and determine a position and orientation of thesensors 604, 608 in relation to the known electromagnetic field emittedby the electromagnetic filed emitter 602.

In one or more embodiments, a mapping database 632 may be consulted todetermine the location coordinates of the sensors 604, 608. The mappingdatabase 632 may reside in the belt pack 620 in some embodiments. In theillustrated embodiment, the mapping database 632 resides on a cloudresource 630. As shown in FIG. 6, the computing apparatus 607communicates wirelessly to the cloud resource 630. The determined poseinformation in conjunction with points and images collected by the ARsystem may then be communicated to the cloud resource 630, and then beadded to the passable world model 634.

As described above, conventional electromagnetic emitters may be toobulky for use in AR devices. Therefore, the electromagnetic fieldemitter may be engineered to be compact, using smaller coils compared totraditional systems. However, given that the strength of theelectromagnetic field decreases as a cubic function of the distance awayfrom the field emitter, a shorter radius between the electromagneticsensors 604 and the electromagnetic field emitter 602 (e.g., about 3-3.5ft.) may reduce power consumption while maintaining acceptable fieldstrength when compared to conventional systems such as the one detailedin FIG. 4.

In one or more embodiments, this feature may be utilized to prolong thelife of the battery 610 that powers the controller 606 and theelectromagnetic field emitter 602. Alternatively, this feature may beutilized to reduce the size of the coils generating the magnetic fieldat the electromagnetic field emitter 602. However, in order to get thesame strength of magnetic field, the power of the electromagnetic fieldemitter 602 may be need to be increased. This allows for anelectromagnetic field emitter unit 602 that may fit compactly at thehand-held controller 606.

Several other changes may be made when using the electromagnetictracking system for AR devices. In one or more embodiments, IMU-basedpose tracking may be used. In such embodiments, maintaining the IMUs asstable as possible increases an efficiency of the pose detectionprocess. The IMUs may be engineered such that they remain stable up to50-100 milliseconds, which results in stable signals with poseupdate/reporting rates of 10-20 Hz. It should be appreciated that someembodiments may utilize an outside pose estimator module (because IMUsmay drift over time) that may enable pose updates to be reported at arate of 10-20 Hz. By keeping the IMUs stable for a reasonable amount oftime, the rate of pose updates may be dramatically decreased to 10-20 Hz(as compared to higher frequencies in conventional systems).

Yet another way to conserve power of the AR system may be to run theelectromagnetic tracking system at a 10% duty cycle (e.g., only pingingfor ground every 100 milliseconds). In other words, the electromagnetictracking system operates for 10 milliseconds out of every 100milliseconds to generate a pose estimate. This directly translates topower savings, which may, in turn, affect size, battery life and cost ofthe AR device.

In one or more embodiments, this reduction in duty cycle may bestrategically utilized by providing two hand-held controllers (notshown) rather than just one. For example, the user may be playing a gamethat requires two totems, etc. Or, in a multi-user game, two users mayhave their own totems/hand-held controllers to play the game. When twocontrollers (e.g., symmetrical controllers for each hand) are usedrather than one, the controllers may operate at offset duty cycles. Thesame concept may also be applied to controllers utilized by twodifferent users playing a multi-player game, for example.

Referring now to FIG. 7, an example flow chart describing theelectromagnetic tracking system in the context of AR devices isdescribed. At 702, the hand-held controller 606 emits a magnetic field.At 704, the electromagnetic sensors 604 (placed on headset 58, belt pack620, etc.) detect the magnetic field. At 706, a position and orientationof the headset/belt is determined based on a behavior of the coils/IMUs608 at the sensors 604. In some embodiments, the detected behavior ofthe sensors 604 is communicated to the computing apparatus 607, which inturn determines the position and orientation of the sensors 604 inrelation to the electromagnetic field(e.g., coordinates relative to thehand-held component). Of course, it should be appreciated that thesecoordinates may then be converted to world coordinates, since the headpose relative to the world may be known through SLAM processing, asdiscussed above.

At 708, the pose information is conveyed to the computing apparatus 607(e.g., at the belt pack 620 or headset 58). At 710, optionally, thepassable world model 634 may be consulted determine virtual content tobe displayed to the user based on the determined head pose and handpose. At 712, virtual content may be delivered to the user at the ARheadset 58 based on the correlation. It should be appreciated that theflowchart described above is for illustrative purposes only, and shouldnot be read as limiting.

Advantageously, using an electromagnetic tracking system similar to theone outlined in FIG. 6 enables pose tracking at a higher refresh rateand lower latency (e.g., head position and orientation, position andorientation of totems, and other controllers). This allows the AR systemto project virtual content with a higher degree of accuracy, and withlower latency when compared to optical tracking techniques forcalculating pose information.

Referring to FIG. 8, a system configuration is illustrated featuringmany sensing components, similar to the sensors described above. Itshould be appreciated that the reference numbers of FIGS. 2A-2D, andFIG. 6 are repeated in FIG. 8. A head mounted wearable component 58 isshown operatively coupled 68 to a local processing and data module 70,such as a belt pack (similar to FIG. 2D), here using a physicalmulticore lead which also features a control and quick release module 86as described below in reference to FIGS. 9A-9F. The local processing anddata module 70 may be operatively coupled 100 to a hand held component606 (similar to FIG. 6). In one or more embodiments, the localprocessing module 70 may be coupled to the hand-held component 606through a wireless connection such as low power Bluetooth®. In one ormore embodiments, the hand held component 606 may also be operativelycoupled 94 directly to the head mounted wearable component 58, such asby a wireless connection such as low power Bluetooth®.

Generally, where IMU data is passed in order to detect pose informationof various components, a high-frequency connection may be desirable,such as in the range of hundreds or thousands of cycles/second orhigher. On the other hand, tens of cycles per second may be adequate forelectromagnetic localization sensing, such as by the sensor 604 andtransmitter 602 pairings. Also shown is a global coordinate system 10,representative of fixed objects in the real world around the user, suchas a wall 8. Cloud resources 46 also may be operatively coupled 42, 40,88, 90 to the local processing and data module 70, to the head mountedwearable component 58, to resources which may be coupled to the wall 8or other item fixed relative to the global coordinate system 10,respectively. The resources coupled to the wall 8 or having knownpositions and/or orientations relative to the global coordinate system10 may include a Wi-Fi transceiver 114, an electromagnetic emitter 602and/or receiver 604, a beacon or reflector 112 configured to emit orreflect a given type of radiation, such as an infrared LED beacon, acellular network transceiver 110, a RADAR emitter or detector 108, aLIDAR emitter or detector 106, a GPS transceiver 118, a poster or markerhaving a known detectable pattern 122, and a camera 124.

The head mounted wearable component 58 features similar components, asillustrated, in addition to lighting emitters 130 configured to assistthe camera 124 detectors, such as infrared emitters 130 for an infraredcamera 124. In one or more embodiments, the head mounted wearablecomponent 58 may further comprise one or more strain gauges 116, whichmay be fixedly coupled to the frame or mechanical platform of the headmounted wearable component 58 and configured to determine deflection ofsuch platform in between components such as electromagnetic receiversensors 604 or display elements 62, wherein it may be valuable tounderstand if bending of the platform has occurred, such as at a thinnedportion of the platform, such as the portion above the nose on theeyeglasses-like platform depicted in FIG. 8.

The head mounted wearable component 58 may also include a processor 128and one or more IMUs 102. Each of the components preferably areoperatively coupled to the processor 128. The hand held component 606and local processing and data module 70 are illustrated featuringsimilar components. As shown in FIG. 8, with so many sensing andconnectivity means, such a system is likely to be heavy, large,relatively expensive, and likely to consume large amounts of power.However, for illustrative purposes, such a system may be utilized toprovide a very high level of connectivity, system component integration,and position/orientation tracking. For example, with such aconfiguration, the various main mobile components (58, 70, 606) may belocalized in terms of position relative to the global coordinate systemusing Wi-Fi, GPS, or Cellular signal triangulation; beacons,electromagnetic tracking (as described above), RADAR, and LIDIR systemsmay provide yet further location and/or orientation information andfeedback. Markers and cameras also may be utilized to provide furtherinformation regarding relative and absolute position and orientation.For example, the various camera components 124, such as those showncoupled to the head mounted wearable component 58, may be utilized tocapture data which may be utilized in simultaneous localization andmapping protocols, or “SLAM”, to determine where the component 58 is andhow it is oriented relative to other components.

Referring to FIGS. 9A-9F, various aspects of the control and quickrelease module 86 are depicted. Referring to FIG. 9A, two outer housing134 components are coupled together using a magnetic couplingconfiguration which may be enhanced with mechanical latching. Buttons136 for operation of the associated system may be included. FIG. 9Billustrates a partial cutaway view with the buttons 136 and underlyingtop printed circuit board 138 shown. Referring to FIG. 9C, with thebuttons 136 and underlying top printed circuit board 138 removed, afemale contact pin array 140 is visible. Referring to FIG. 9D, with anopposite portion of housing 134 removed, the lower printed circuit board142 is visible. With the lower printed circuit board 142 removed, asshown in FIG. 9E, a male contact pin array 144 is visible.

Referring to the cross-sectional view of FIG. 9F, at least one of themale pins or female pins are configured to be spring-loaded such thatthey may be depressed along each pin's longitudinal axis. In one or moreembodiments, the pins may be termed “pogo pins” and may generallycomprise a highly conductive material, such as copper or gold. Whenassembled, the illustrated configuration may mate 46 male pins withfemale pins, and the entire assembly may be quick-release decoupled inhalf by manually pulling it apart and overcoming a magnetic interface146 load which may be developed using north and south magnets orientedaround the perimeters of the pin arrays 140, 144.

In one embodiment, an approximate 2 kg load from compressing the 46 pogopins is countered with a closure maintenance force of about 4 kg. Thepins in the arrays 140, 144 may be separated by about 1.3 mm, and thepins may be operatively coupled to conductive lines of various types,such as twisted pairs or other combinations to support USB 3.0, HDMI2.0, I2S signals, GPIO, and MIPI configurations, and high current analoglines and grounds configured for up to about 4 amps/5 volts in oneembodiment.

Referring to FIG. 10, it is helpful to have a minimizedcomponent/feature set to be able to minimize the weight and bulk of thevarious components, and to arrive at a relatively slim head mountedcomponent, for example, such as that of head mounted component 58featured in FIG. 10. Thus various permutations and combinations of thevarious components shown in FIG. 8 may be utilized.

Referring to FIG. 11A, an electromagnetic sensing coil assembly (604,e.g., 3 individual coils coupled to a housing) is shown coupled to ahead mounted component 58. Such a configuration adds additional geometry(i.e., a protrusion) to the overall assembly which may not be desirable.Referring to FIG. 11B, rather than housing the coils in a box or singlehousing as in the configuration of FIG. 11A, the individual coils may beintegrated into the various structures of the head mounted component 58,as shown in FIG. 11B. For example, x-axis coil 148 may be placed in oneportion of the head mounted component 58 (e.g., the center of theframe). Similarly, the y-axis coil 150 may be placed in another portionof the head mounted component 58 (e.g., either bottom side of theframe). Similarly, the z-axis coil 152 may be placed in yet anotherportion of the head mounted component 58 (e.g., either top side of theframe).

FIGS. 12A-12E illustrate various configurations for featuring a ferritecore coupled to an electromagnetic sensor to increase field sensitivity.Referring to FIG. 12A, the ferrite core may be a solid cube 1202.Although the solid cube may be most effective in increasing fieldsensitivity, it may also be the most heavy when compared to theremaining configurations depicted in FIGS. 12B-12E. Referring to FIG.12B, a plurality of ferrite disks 1204 may be coupled to theelectromagnetic sensor. Similarly, referring to FIG. 12C, a solid cubewith a one axis air core 1206 may be coupled to the electromagneticsensor. As shown in FIG. 12C, an open space (i.e., the air core) may beformed in the solid cube along one axis. This may decrease the weight ofthe cube, while still providing the necessary field sensitivity. In yetanother embodiment, referring to FIG. 12D, a solid cube with a threeaxis air core 1208 may be coupled to the electromagnetic sensor. In thisconfiguration, the solid cube is hollowed out along all three axes,thereby decreasing the weight of the cube considerably. Referring toFIG. 12E, ferrite rods with plastic housing 1210 may also be coupled tothe electromagnetic sensor. It should be appreciated that theembodiments of FIGS. 12B-12E are lighter in weight than the solid coreconfiguration of FIG. 12A and may be utilized to save mass, as discussedabove.

Referring to FIGS. 13A-13C, time division multiplexing (“TDM”) may beutilized to save mass as well. For example, referring to FIG. 13A, aconventional local data processing configuration is shown for a 3-coilelectromagnetic receiver sensor, wherein analog currents come in fromeach of the X, Y, and Z coils (1302, 1304 and 1306), go into a separatepre-amplifier 1308, go into a separate band pass filter 1310, a separatepre-amplifier 1312, through an analog-to-digital converter 1314, andultimately to a digital signal processor 1316.

Referring to the transmitter configuration of FIG. 13B, and the receiverconfiguration of FIG. 13C, time division multiplexing may be utilized toshare hardware, such that each coil sensor chain doesn't require its ownamplifiers, etc. This may be achieved through a TDM switch 1320, asshown in FIG. 13B, which facilitates processing of signals to and frommultiple transmitters and receivers using the same set of hardwarecomponents (amplifiers, etc.). In addition to removing sensor housings,and multiplexing to save on hardware overhead, signal to noise ratiosmay be increased by having more than one set of electromagnetic sensors,each set being relatively small relative to a single larger coil set.Also, the low-side frequency limits, which generally are needed to havemultiple sensing coils in close proximity, may be improved to facilitatebandwidth requirement improvements. It should be noted that there may bea tradeoff with multiplexing, in that multiplexing generally spreads outthe reception of radiofrequency signals in time, which results ingenerally coarser signals. Thus, larger coil diameters may be requiredfor multiplexed systems. For example, where a multiplexed system mayrequire a 9 mm-side dimension cubic coil sensor box, a non-multiplexedsystem may only require a 7 mm-side dimension cubic coil box for similarperformance. Thus, it should be noted that there may be tradeoffs inminimizing geometry and mass.

In another embodiment wherein a particular system component, such as ahead mounted component 58 features two or more electromagnetic coilsensor sets, the system may be configured to selectively utilize thesensor and electromagnetic emitter pairing that are closest to eachother to optimize the performance of the system.

Referring to FIG. 14, in one embodiment, after a user powers up his orher wearable computing system 160, a head mounted component assembly maycapture a combination of IMU and camera data (the camera data beingused, for example, for SLAM analysis, such as at the belt pack processorwhere there may be more RAW processing horsepower present) to determineand update head pose (i.e., position and orientation) relative to a realworld global coordinate system 162. The user may also activate ahandheld component to, for example, play an augmented reality game 164,and the handheld component may comprise an electromagnetic transmitteroperatively coupled to one or both of the belt pack and head mountedcomponent 166. One or more electromagnetic field coil receiver sets(e.g., a set being 3 differently-oriented individual coils) coupled tothe head mounted component may be used to capture magnetic flux from theelectromagnetic transmitter. This captured magnetic flux may be utilizedto determine positional or orientational difference (or “delta”),between the head mounted component and handheld component 168.

In one or more embodiments, the combination of the head mountedcomponent assisting in determining pose relative to the globalcoordinate system, and the hand held assisting in determining relativelocation and orientation of the handheld relative to the head mountedcomponent, allows the system to generally determine where each componentis located relative to the global coordinate system, and thus the user'shead pose, and handheld pose may be tracked, preferably at relativelylow latency, for presentation of augmented reality image features andinteraction using movements and rotations of the handheld component 170.

Referring to FIG. 15, an embodiment is illustrated that is somewhatsimilar to that of FIG. 14, with the exception that the system has manymore sensing devices and configurations available to assist indetermining pose of both the head mounted component 172 and a hand heldcomponent 176, 178, such that the user's head pose, and handheld posemay be tracked, preferably at relatively low latency, for presentationof augmented reality image features and interaction using movements androtations of the handheld component 180.

Specifically, after a user powers up his or her wearable computingsystem 160, a head mounted component captures a combination of IMU andcamera data for SLAM analysis in order to determined and update headpose relative a real-world global coordinate system. The system may befurther configured to detect presence of other localization resources inthe environment, like Wi-Fi, cellular, beacons, RADAR, LIDAR, GPS,markers, and/or other cameras which may be tied to various aspects ofthe global coordinate system, or to one or more movable components 172.

The user may also activate a handheld component to, for example, play anaugmented reality game 174, and the handheld component may comprise anelectromagnetic transmitter operatively coupled to one or both of thebelt pack and head mounted component 176. Other localization resourcesmay also be similarly utilized. One or more electromagnetic field coilreceiver sets (e.g., a set being 3 differently-oriented individualcoils) coupled to the head mounted component may be used to capturemagnetic flux from the electromagnetic transmitter. This capturedmagnetic flux may be utilized to determine positional or orientationaldifference (or “delta”), between the head mounted component and handheldcomponent 178.

Thus, the user's head pose and the handheld pose may be tracked atrelatively low latency for presentation of AR content and/or forinteraction with the AR system using movement or rotations of thehandheld component 180.

Various exemplary embodiments of the invention are described herein.Reference is made to these examples in a non-limiting sense. They areprovided to illustrate more broadly applicable aspects of the invention.Various changes may be made to the invention described and equivalentsmay be substituted without departing from the true spirit and scope ofthe invention. In addition, many modifications may be made to adapt aparticular situation, material, composition of matter, process, processact(s) or step(s) to the objective(s), spirit or scope of the presentinvention. Further, as will be appreciated by those with skill in theart that each of the individual variations described and illustratedherein has discrete components and features which may be readilyseparated from or combined with the features of any of the other severalembodiments without departing from the scope or spirit of the presentinventions. All such modifications are intended to be within the scopeof claims associated with this disclosure.

The invention includes methods that may be performed using the subjectdevices. The methods may comprise the act of providing such a suitabledevice. Such provision may be performed by the end user. In other words,the “providing” act merely requires the end user obtain, access,approach, position, set-up, activate, power-up or otherwise act toprovide the requisite device in the subject method. Methods recitedherein may be carried out in any order of the recited events which islogically possible, as well as in the recited order of events.

Exemplary aspects of the invention, together with details regardingmaterial selection and manufacture have been set forth above. As forother details of the present invention, these may be appreciated inconnection with the above-referenced patents and publications as well asgenerally known or appreciated by those with skill in the art. The samemay hold true with respect to method-based aspects of the invention interms of additional acts as commonly or logically employed.

In addition, though the invention has been described in reference toseveral examples optionally incorporating various features, theinvention is not to be limited to that which is described or indicatedas contemplated with respect to each variation of the invention. Variouschanges may be made to the invention described and equivalents (whetherrecited herein or not included for the sake of some brevity) may besubstituted without departing from the true spirit and scope of theinvention. In addition, where a range of values is provided, it isunderstood that every intervening value, between the upper and lowerlimit of that range and any other stated or intervening value in thatstated range, is encompassed within the invention.

Also, it is contemplated that any optional feature of the inventivevariations described may be set forth and claimed independently, or incombination with any one or more of the features described herein.Reference to a singular item, includes the possibility that there areplural of the same items present. More specifically, as used herein andin claims associated hereto, the singular forms “a,” “an,” “said,” and“the” include plural referents unless the specifically stated otherwise.In other words, use of the articles allow for “at least one” of thesubject item in the description above as well as claims associated withthis disclosure. It is further noted that such claims may be drafted toexclude any optional element. As such, this statement is intended toserve as antecedent basis for use of such exclusive terminology as“solely,” “only” and the like in connection with the recitation of claimelements, or use of a “negative” limitation.

Without the use of such exclusive terminology, the term “comprising” inclaims associated with this disclosure shall allow for the inclusion ofany additional element--irrespective of whether a given number ofelements are enumerated in such claims, or the addition of a featurecould be regarded as transforming the nature of an element set forth insuch claims. Except as specifically defined herein, all technical andscientific terms used herein are to be given as broad a commonlyunderstood meaning as possible while maintaining claim validity.

The breadth of the present invention is not to be limited to theexamples provided and/or the subject specification, but rather only bythe scope of claim language associated with this disclosure.

What is claimed is:
 1. A method to display augmented reality,comprising: emitting, through an electromagnetic field emitter, a knownmagnetic field; measuring, through an electromagnetic sensor, aparameter related to a magnetic flux at the electromagnetic sensor as aresult of the emitted known magnetic field; determining pose informationrelative to the electromagnetic field emitter based at least in part onthe measured parameter related to the magnetic flux at theelectromagnetic sensor; capturing image data through one or more camerascoupled to a head-mounted component; determining head pose informationutilizing the image data; and displaying virtual content on an ARdisplay system to a user based at least in part on both the determinedpose information relative to the electromagnetic field emitter and thehead pose information.
 2. The method of claim 1, wherein: theelectromagnetic sensor comprises at least three coils to measuremagnetic flux in three directions; and the at least three coils areintegrated into the head-mounted component at different locations of thehead-mounted component of the AR display system.
 3. The method of claim1, wherein determining head pose information utilizing the image data isdetermined based at least in part on a SLAM analysis utilizing the imagedata.
 4. The method of claim 1, wherein the electromagnetic fieldemitter resides in a mobile component of the AR display system.
 5. Themethod of claim 4, wherein the mobile component is a hand-heldcomponent.
 6. The method of claim 4, wherein the mobile component is atotem.
 7. The method of claim 4, wherein the mobile component is ahead-mounted component of the AR display system.
 8. The method of claim1, wherein the AR display system is housed in the head-mountedcomponent, and the electromagnetic sensor is operatively coupled to thehead-mounted component.
 9. The method of claim 1, further comprising:determining world coordinates of the electromagnetic sensor based atleast in part on the SLAM analysis of the image data utilized todetermine head pose information.
 10. The method of claim 1, wherein theelectromagnetic sensors comprise one or more inertial measurement units(IMUs).
 11. The method of claim 1, wherein the pose informationcorresponds to at least a position and orientation of theelectromagnetic field emitter relative to the world, and the head poseinformation corresponds to at least a location and orientation of theuser's head relative to the world.
 12. The method of claim 1, whereinthe pose information is analyzed to determine world coordinatescorresponding to the electromagnetic field emitter.
 13. The method ofclaim 1, further comprising detecting an interaction with one or morevirtual contents based at least in part on the pose informationcorresponding to the electromagnetic field emitter.
 14. The method ofclaim 13, further comprising displaying virtual content to the userbased at least in part on the detected interaction.
 15. The method ofclaim 1, further comprising decoupling the magnetic field emitted by theelectromagnetic field emitter through a control and quick releasemodule.
 16. The method of claim 1, further comprising determining worldcoordinates of the electromagnetic field emitter through localizationresources other than the electromagnetic sensor, electromagnetic fieldemitter and image data.
 17. The method of claim 16, wherein theadditional localization resources comprises a GPS receiver.
 18. Themethod of claim 16, wherein the additional localization resourcescomprises a beacon.
 19. The method of claim 1, wherein theelectromagnetic sensor comprises at least one of: a stack of ferritedisks; a plurality of ferrite rods each having a polymer coating; and atime division multiplexing switch.